Skip to content

feat(cli): score a file or stdin of requests in one batch - #511

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:feat-cli-batch
Sep 27, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:feat-cli-batch

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

Why

pip install laya already puts a laya command on the path for quick local testing, but it can
only score one text per process: laya "text" --predict calls Router.predict once, so trying
a set of tickets means one process — one checkpoint load, one forward pass — per ticket. The
library has had Router.predict_batch / route_batch for exactly this since #294-era batching
landed; the CLI never exposed them. This closes the gap between what the package can do and what
typing laya at a shell gets you.

What

  • laya/cli.py — --batch FILE (- for stdin): one request per non-blank line, scored through
    a single Router.predict_batch call (or route_batch for the no---predict routing mode).
    Existing semantics carry over unchanged: --predict / --preset / --model / --lang /
    --task, the per-preset state keys, and --json — which in batch mode prints JSONL, one
    record per line, matching how laya-evals consumes datasets. --batch-size N bounds the
    forward pass; blank lines are skipped; missing/empty files and text + --batch exit 2 with a
    named error instead of a traceback.
  • tests/test_cli.py — 14 new checks, still weight-free (stubbed Router): one batch call not N
    single calls, line skipping/stripping, --batch-size and per-request flag forwarding, preset
    state keys, JSONL output, routing mode via route_batch, and the three error paths.
  • README — two example lines and the measured note in the Command line section.

Measured

Real convaiinnovations/laya checkpoint, 20 English support tickets, one warm process, CPU
(M3), best of 3:

how you score 20 tickets wall clock per state vs loop
loop predict (what the CLI could do before) 2.6 s 130 ms 1.0x
--batch --batch-size 8 1.0 s 52 ms ~2.5x
--batch (single pass) 1.0 s ~50 ms ~2.6x

Decision parity: 0/20 flips (choice label / noul>=0.5 / round(score)) and max choice
probability drift 0.0 versus the per-state loop. At the process level a full batch also amortizes
checkpoint load: for t in tickets: laya --predict pays it once per ticket, laya --batch once
per file.

Notes

Test plan

  • python tests/test_cli.py — 59/59 (45 pre-existing + 14 new)
  • ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120, compileall
  • python tests/test_router.py / tests/test_hooks_api.py — no regressions
  • real-checkpoint A/B measurement above (0 decision flips)
  • CI green on this PR

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254, merged. After #542 landed, --batch also takes --questions, --max-len and --head-max-len, and the budget rides on each request since #583. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants